XG Boost in Mental Health Classification

Introduction

eXtreme Gradient Boosting (XGB) algorithm

• A supervised machine learning algorithm
• Modification of Gradient Boosting Framework
• Ensemble of weak decision trees
• L1,L2 regularization
• High performance, speed, scalability
• Applications in healthcare, education, public health, finance, and engineering.

Introduction

XGBoost for mental disorder classification

• Using XGBoost on clinical and survey data
• Classification of individuals into one of four categories: Bipolar I Disorder, Bipolar II Disorder, Major Depressive Disorder, and Normal
• XGBoosts proves to be a suitable algorithm for the problem

Literature Review

eXtreme Gradient Boosting (XGBoost) in healthcare

• XGBoost strengths in real world problems handling class imbalance, heterogeneous data types, or non-linear relationships. (Chen and Guestrin 2016)
• XGBoost combined with DL for breast cancer classification with high reliability. (Liew, Hameed, and Clos 2021)
• XGBoost with biomarker data to improve depression diagnoses in a large Dutch population dataset.(Sharma and Verbeke 2020)
• XGBoost to multi-modal datasets to predict self-harm in young adults.(Xu et al. 2024)
• A hybrid algorithm (XGBoost-HOA) to classify depression, anxiety, and stress
• Dual XGBoost models applied to distinguish between deficit and non-deficit schizophrenia subtypes using fMRI features.(Zhang, Jia, and Shang 2022)
• XGBoost compared with linear regression in predicting depression among refugee children.(Saleh et al. 2024)

Literature Review

XGBoost limitations for imbalance data

• handled by optimization techniques (Zhang, Jia, and Shang 2022)
• Handled by balancing the class distribution (Sharma and Verbeke 2020)
• Preprocessing and hyperparameter tuning importance in imbalance data for accurate XGBoost algorithm.

Literature Review

XGBoost in Education

• XGBoost in academic prediction models (Hu and Song 2019)
• XGBoost out performing traditional logistic regression models in predicting learner performance (Hakkal and Ait Lahcen 2024).

XGBoost in Public health

• XGBoost used to predict daily COVID-19 cases in the United States.(Fang et al. 2022)
• A hybrid model integrating XGBoost, Random Forest, and Antlion Optimization used to predict infectious disease outbreaks.(Sivakumar and Elangovan 2023)

Literature Review

XGBoost in financial sector

• XGBoost model’s ability to rank feature importance and prevent overfitting using regularization made it a top performer in credit risk prediction tasks(Li et al. 2020)
• XGBoost to forecast volatility in the U.S. stock market, identifying the Economic Policy Uncertainty Index as a critical predictor.(Fomunyam 2023)

XGBoost In pharmaceutical research

• Application of XGBoost in drug development.(Wiens et al. 2025)

Literature Review

XGBoost in educational diagnostics

• XGBoost enhanced learner performance prediction and knowledge tracing on platforms like ASSIST09 and Algebra08. (Su et al. 2023) and (Hakkal and Ait Lahcen 2024)

XGBoost in Sports Analytics

• XGBoost used to predict ultramarathon running speeds based on demographics and environmental factors.(Nikolaidis, Knechtle, and co-authors 2023)

Methods: Extreme Gradient Boosting (XGBoost)

• Ensemble of weak decision trees
• Trained sequentially, each tree corrects previous errors
• Captures non-linear relationships within the 17-variable dataset
• Uses gradient descent to minimize loss
• Includes regularization to prevent overfitting

Methods: XGBoost Formulas

• Loss term: Measures prediction error
• Regularization term: Penalizes model complexity
• Ensures balance between accuracy and generalization

\[ \text{Obj} = \sum_{i=1}^{n} L(y_i, \hat{y}_i) + \sum_{k=1}^{K} \Omega(f_k) \]

Data Exploration and Visualization

• Sourced from Kaggle (2023 clinical records)
• 120 individuals assessed for mental disorders -Labels: -Normal -Bipolar Type I -Bipolar Type II -Depression
• 17 diagnostic variables
• Mix of clinical, behavioral, demographic factors

Data Exploration and Visualization

• Examined class distribution
• Identified potential imbalance or outliers
• Explored correlations between features

Modeling and Results

Data Preprocessing

To facilitate analysis, all categorical variables in the dataset were transformed into numerical or grouped categorical formats in Excel. The transformations were applied as follows:

• Ordinal responses (Sadness, Euphoric, Exhausted) recoded:
  • Seldom=1, Sometimes=2, Usually=3, Most-Often=4
• Binary items (Suicidal Thoughts, Mood Swing):
  • Yes=1, No=0
• 1–10 rating scales (Sexual Activity, Optimism) grouped:
  • 1–3 → Cat 1, 4–6 → Cat 2, 7–9 → Cat 3

Modeling and Results

Hyperparameter Optimization

• The dataset was randomly partitioned into an 80/20 training–testing split.
  • This split resulted in 96 patient surveys used for model training and 24 surveys reserved for independent testing.
  • The predictive target for all models was the expert clinical diagnosis
• An XGBoost model specification was created using the tidymodels framework where all major hyperparameters were set to be tuned including: - the number of trees - tree depth - learning rate - loss reduction, - minimum node size - subsampling rate - number of predictors considered at each split

Modeling and Results

Hyperparameter Optimization

• Five-fold stratified cross-validation was used for hyperparameter tuning.
  • Where the training dataset was partitioned into five equally sized folds
  • For each tuning iteration:
    • Models were trained on four folds
    • Validated on the remaining fold
    • Which ensured stable and unbiased performance estimates

(singh2024crossvalidation_image?)

Modeling and Results

Hyperparameter Optimization

• The final hyperparameter grid consists of 30 candidate hyperparameter combination which was constructed using a space-filling design.
  • Model performance was assessed using accuracy and the area under the ROC curve (ROC AUC).
  • Visualizing the tuning results revealed the ROC AUC versus number of trees plot indicated that the optimal tree depth was approximately 2, suggesting that relatively shallow trees generalized best for this dataset.
  • Model performance increased steadily with the number of boosting iterations, with the highest ROC AUC occurring at around 625 trees. At this configuration, the model achieved an accuracy exceeding 0.80 and a ROC AUC greater than 0.90, indicating strong discriminatory performance.

VisulizationTuningResults

Modeling and Results

Hyperparameter Optimization

• The select_best() function from the tidymodels framework was used to systematically extract the hyperparameter combination that achieved the highest ROC AUC across all resampled evaluations.

• The resulting optimal configuration included 3 predictors (mtry = 3), 627 boosting iterations, a minimum node size (min_n) of 2, a learning rate of 0.16501, a loss-reduction parameter of 0.0034, and a subsample proportion of 0.5655.

  • This combination represents the tuning grid entry labeled pre0_mod06_post0, which achieved the best balance between model complexity and predictive performance during cross-validation.

BestParams

Modeling and Results

Model Evaluation & Performance

Prediction

The model’s predictions on the held-out test set showed strong agreement with the expert diagnoses. Specifically:

• Bipolar Type-1: 4 of 6 cases were correctly predicted (66.7%)
  
• Bipolar Type-2: 5 of 7 cases were correctly predicted (71.4%)
  
• Depression: 6 of 7 cases were correctly predicted (85.7%)
  
• Normal: 4 of 6 cases were correctly predicted (66.7%)

Modeling and Results

Model Evaluation & Performance

The XGBoost multiclass classification model demonstrates moderate overall predictive performance, with room for improvement but also several encouraging indicators of discriminative ability. - Accuracy: 0.654
An accuracy of 65.4% suggests that the model correctly classifies roughly two-thirds of the observations. - Kappa: 0.534
The Kappa value of 0.534 reflects moderate agreement between the model’s predictions and the true class labels after accounting for chance. - ROC AUC (macro-weighted): 0.927
The macro-weighted ROC AUC of 0.927 suggests that the model does a strong job of ranking observations correctly across all classes, even if it occasionally makes incorrect final class assignments.

Taken together, the metrics indicate that the model has strong underlying discriminative power (high AUC) but moderate final classification performance (accuracy and kappa).

Modeling and Results

Feature Importance & Interepretability

Modeling and Results

Sensitvity & Robustness Analysis

Modeling and Results

Summary of Modeling Results

Conclusion: Study Purpose

• Evaluate effectiveness of XGBoost for predicting mental health outcomes.
• Develop, tune, and validate the model.
• Identify highest‑predictive features and assess ability to capture non‑linear relationships.
• Model showed strong accuracy and generalizability.

Conclusion: Key Findings

• Top predictors: stress indicators, behavioral patterns, wellness factors.
• High feature importance suggests relevance to mental‑health risk.
• Model effectively distinguished between classes.
• Stable performance across tuning configurations with minimal overfitting.

Conclusion: Implications

• XGBoost shows potential for early warning mental‑health monitoring.
• Useful for identifying at‑risk individuals based on survey data.
• Self‑reported data may introduce bias—interpret with care.

Conclusion: Future Work

• Use larger and more diverse datasets.
• Incorporate behavioral or time‑based variables.
• Compare with Random Forest, LightGBM, neural networks.
• Test in real‑world settings.
• Address fairness and ethical use of health data.

Conclusion: Overview

• XGBoost is effective for survey‑based mental‑health prediction.
• Provides fast and interpretable insights alongside traditional assessments.
• Predictive modeling could expand tools for mental‑health research and practice.